Multimodal Co-located Collaboration

1 This work was done in collaboration with Mitsubishi Electric Research Laboratories in Cambridge, Massachusetts ABSTRACT People naturally perform multimodal interactions in everyday real world settings, as they collaborate over large visual surfaces such as paper maps on walls and tables. While a new generation of research technologies now supports co-located collaboration, they do not yet directly leverage such rich multimodal interaction. Even though a rich behavioural foundation is emerging that informs the design of co-located collaborative technologies, most systems still typically limit input to a single finger or pointer. This problem is partly caused by the fact that single point interaction is the only input easily accessible to researchers through existing toolkits and input APIs. In this research, I distil existing theories, models and ethnographic studies on co-present collaboration into behavioural foundations that describe the individual and group benefits for using gesture and speech multimodal input in a co-located large display setting. Next, I will develop a toolkit that facilitates rapid prototyping of multimodal co-located applications over large digital displays. Finally, using these applications, I will conduct a number of studies exploring design in a multimodal setting.